Application of reinforcement learning to control a multi-agent system

Abstract: This study takes place in the context of multi-agent systems (MAS), and especially reactive ones. In such a system, interactions are essential, and trigger a collective behaviour that is not directly linked to the individual ones. Whereas the evolution of the system is unknown if not tried, the regularity of emergent structures in the system is observable and forms a global behaviour. In this paper, we propose to control the global behaviour of a MAS thanks to reinforcement learning tools applied at its global level. We also highlight the choice of the features taken into account to achieve this control, that is the information considered to decide which action to perform

Mod\'elisation multi-niveaux dans AA4MM

In this article, we propose to represent a multi-level phenomenon as a set of interacting models. This perspective makes the levels of representation and their relationships explicit. To deal with coherence, causality and coordination issues between models, we rely on AA4MM, a metamodel dedicated to such a representation. We illustrate our proposal and we show the interest of our approach on a flocking phenomenon

Controlling the Global Behaviour of a Reactive MAS : Reinforcement Learning Tools

International audienceReactive multi-agent systems present global behaviours uneasily linked to their local dynamics. When it comes to controlling such a system, usual analytical tools are difficult to use so specific techniques have to be engineered. We propose an experimental dynamical approach to control the global behaviour of a reactive multi-agent system. We use reinforcement learning tools to link global information of the system to control actions. We propose to use the behaviour of the system as this global information. The controllability is evaluated in terms of rate of convergence towards a target behaviour. We compare the results obtained on a toy example with the usual approach of parameter setting

Bio-inspired Mechanisms for Artificial Self-organised Systems

Research on self-organization tries to describe and explain forms, complex patterns and behaviours that arise from a collection of entities without an external organizer. As researchers in artificial systems, our aim is not to mimic self-organizing phenomena arising in Nature, but to understand and to control underlying mechanisms allowing desired emergence of forms, complex patterns and behaviours. In this paper we analyze three forms of self-organization: stigmergy, reinforcement mechanisms and cooperation. For each forms of self-organisation, we present a case study to show how we transposed it to some artificial systems and then analyse the strengths and weaknesses of such an approach

A Nonlinear Multi-agent System designed for Swarm Intelligence : the Logistic MAS

International audienceAnt algorithms and flocking algorithms are the two main programming paradigms in swarm intelligence. They are built on stochastic models, widely used in optimization problems. However, though this modeling leads to high-performance algorithms, some mechanisms, like the symmetry break in ant decision, are still not well understood at the local ant level. Moreover, there is currently no modeling approach which joins the two paradigms. This paper proposes an entirely novel approach to the mathematical foundations of swarm algorithms: contrary to the current stochastic approaches, we show that an alternative deterministic model exists, which has its origin in deterministic chaos theory. We establish a reactive multi-agent system, based on logistic nonlinear decision maps, and designed according to the influence-reaction scheme. The rewriting of the decision functions leads to a new way of understanding the swarm phenomena in terms of state synchronization, and enables the analysis of their convergence behavior through bifurcation diagrams. We apply our approach on two concrete examples of each algorithm class, in order to demonstrate its general applicability

Multi-level Modeling as a Society of Interacting Models

We propose to consider a multi-level representation from a multi-modeling point of view. We define a framework to better specify the concepts used in multi-level modeling and their relationships. This framework is implemented through the AA4MM meta-model, which benefits from a middleware layer. This meta-model uses the multi-agent paradigm to consider a multi-model as a society of interacting models. We extend this meta-model to consider multi-level modeling and present a proof of concept of a collective motion example where we show the ability of this approach to rapidly change from one pattern of interaction to another one by reusing some of the meta-model's components

Flocking as a Synchronization Phenomenon with Logistic Agents

International audienceIn this paper, we intend to show that the flocking phenomenon observed in many animal species behaviors, may be modeled as a synchronization process occurring within entity states. Although flocking has been widely studied and simulated in Swarm Intelligence, few works mention synchronization as a key aspect of the problem and model it properly. This paper proposes a modeling in terms of a reactive multi-agent system composed of interacting logistic agents moving in an environment. This specific MAS called Logistic MAS (LMAS) takes actually inspiration from the coupled map lattice field, which provides also many tools to analyse convergence and stability of the system. We develop our approach in both theoretical and applied way to demonstrate its relevance

Study of Self-adaptation Mechanisms in a Swarm of Logistic Agents

International audienceWe are interested in addressing the problem of coordinating a large number of simple agents in order to achieve a given task. Stated in this way, the question leads naturally to the Swarm Intelligence field. In this paper we use a new type of model, directly inspired by Kaneko's coupled map gas model which we have adapted to the multi-agent system paradigm, so as to tackle this generic objective. This model is called a logistic multi-agent system (LMAS): it is composed of reactive situated agents whose individual behavior is governed by a logistic map or more generally a quadratic map. The collective behavior results from couplings between agents and local controls on agents adjusted by local environmental conditions. This way of modelling reveals to enable a wide range of pattern formations and various forms of adaptation to the environment. This paper focuses on the way to design the constitutive mechanisms of LMAS –particularly the perception and action processes– and on the way a self-adaptation process may result from these mechanisms. This study is illustrated with experiments on the predators-prey pursuit problem, in which a set of agents (predators) has to encircle a moving prey. We show that coupling the internal states of agents leads to amplifying the predator aggregation around the prey, whereas altering the internal control variable in each agent through environment perceptions modifies the predator sensitivity to the prey. We finally complete this study by relating the concept of adaptation with concepts of the dynamical system theory: a qualitative dynamical analysis of the capturing process leads to view the prey as a dynamical fixed point of the system

Deterministic Nonlinear Modeling of Ant Algorithm with Logistic Multi-Agent System

International audienceAnt algorithms are one of the main programming paradigms in swarm intelligence. They are built on stochastic decision functions, which can also be found in other types of bio-inspired algorithms with the same mathematical form. However, though this modeling leads to high-performance algorithms, some phenomena, like symmetry break, are still not well understood or modeled at the ant level. This paper proposes an original analysis of the problem : we establish a reactive multi-agent system based on logistic nonlinear decision maps, and designed according to the influence-reaction scheme. Our proposition is an entirely novel approach to the mathematical foundations of ant algorithms : contrary to the current stochastic approaches, we show that an alternative deterministic model exists, which has its origin in deterministic chaos theory. The rewriting of the decision functions leads to a new way of understanding and visualizing the convergence behavior of ant algorithms. We apply our approach on a concrete example, namely the binary bridge problem

Construction de systèmes multi-agents par apprentissage collectif à base d'interactions

National audienceCet article se focalise sur des approches formelles pour la construction de systèmes multi-agents. Ce travail a cherché à proposer des apprentissages décentralisés pour construire les comportements d'agents sociaux. Cet article propose un formalisme original, l'interac-DEC-POMDP inspiré des modèles markoviens au sein duquel les agents peuvent interagir directement et localement entre eux. A partir de ce formalisme, cet article propose aussi un algorithme d'apprentissage décentralisé fondé sur une répartition heuristique des gains des agents au cours des interactions. Une démarche expérimentale a validé sa capacité à produire automatiquement des comportements collectifs. Les techniques présentées pourraient alors constituer des moyens permettant aux agents de décider automatiquement et de manière décentralisée comment s'organiser avec les autres pour résoudre un problème donné